Actions, Policies, Values, and the Basal Ganglia

The basal ganglia are widely believed to be involved in the learned selection of actions. Building on this idea, reinforcement learning (RL) theories of optimal control have had some success in explaining the responses of their key dopaminergic afferents. While these model-free RL theories offer a compelling account of a range of neurophysiological and behavioural data, they offer only an incomplete picture of action control in the brain. Psychologists and behavioural neuroscientists have long appealed to the existence of at least two separate control systems underlying the learned control of behaviour. The habit system is closely identified with the basal ganglia, and we associate it with the model-free RL theories. The other system, more loosely localised in prefrontal regions and without such a detailed theoretical account, is associated with cognitively more sophisticated goal-directed actions. On the critical issue of which system determines the ultimate output when they disagree, there is a wide range of experimental results and sparse theoretical underpinning. Here, we extend the RL account of neural action control by first interpreting goal-directed actions in terms of an alternativemodel-based strategy for RL. Then, by considering the relative uncertainties of modelfree and model-based controllers, we offer a new and more comprehensive account of the confusing experimental results about how the systems trade off control. Our theory offers a more sharply delineated view of the contributions of the basal ganglia to learned behavioural control.